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Abstract 

We  derive  an  explicit  determinantal  formula  for  the  least  squares  (LS) 
solution  of  an  overdetermined  system  of  linear  equations.  From  this  formula 
it  follows  that  the  LS  solution  lies  in  the  convex  hull  S  of  points,  each  of 
which  is  a  solution  of  a  square  subsystem  of  the  whole  system.  The  results 
are  extended  to  weighted  LS  solution;  it  is  shown  that  the  convex  hull  S  in 
which  the  solution  must  lie  is  independent  of  the  weighting  matrix.  r 

Key  words.  Linear  equations,  Least  squares  problems,  Convexity. 
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1.  Introduction 


Consider  the  overdetermined  system  of  linear  equations 

Ax  —  b 


(1.1) 


where  A  is  m  x  n  real  matrix  (m  >  n).  The  Least  squares  solution  of  (1.1) 
is  obtained  from  the  solution  of  the  normal  equation 

ATAx  =  ATb  (1.2) 

The  mxn  system  (1.1)  contains  (  ^  )  squared  n  x  n  subsystems  (obtained 
by  all  possible  choices  of  n  rows  of  A  out  of  m)  which  can  be  indexed  by 
j  €  {1, .  . ^  ™  :=  Pm,n-  Let  yii  be  the  solution  of  the  _7-th  subsystem 

corresponding  to  rows  of  A  indexed  by  Ij  =  {tj,  C  Pmr t,  i.e. 


Ai,yi,  =  bi} 


(1.3) 


The  solution  to  (1.3)  exists  and  is  unique  if  A jy  =  detAi ■.  ^  0. 

In  this  paper  we  obtain  a  new  representation  of  the  least  squares  solution 
i  of  (1.2)  as  follows:  __ 

x=  (14) 

j£Pm.n 

where 

^m,n  =  {j  ^  Pm,n  '■  &Ij  7^  0}  and  A/;  = 

^76 P£.n  ali 

The  A/y’s  are  weights  satisfying:  A/y  >  0  and  J2j^p+n  =  1,  hence  from 
the  representation  (1.4) 


x  €  convex  hull  of  {y/;.  :  j  €  P£  n } 


(1.5) 


We  further  show  that  the  result  (1.5)  remains  true  for  x  being  the  solution 
of  a  weighted  least  squares  problem 

AT  MAx  =  AT  Mb  (1.6) 


where  M  is  a  positive  definite  diagonal  matrix.  For  this  case  each  y jj  is  still 
the  solution  of  (1.3),  independent  of  M.  This  result  is  useful  for  numerical 
methods  whose  ( k  +  1)  st  iterate  xk+i  is  the  solution  of  a  system 

AT  Mk  AXk+i  =  ATMk  b 


(1.7) 
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The  iterative  procedure  (1.7)  is  in  the  heart  of  Karmakar’s  Linear  Program¬ 
ming  method  [4]  and  many  related  methods,  see  e.g.  [3]  and  the  references 
therein.  From  our  result  it  follows  that,  starting  from  any  xo,  the  next 
iteration  points  xi,X2,  -,  all  remain  in  the  fixed  compact  polyhedral  set 
5  =  conv{t //;  :  j  €  P+n}.  The  fact  is  important  for  proving  convergence 
of  the  sequence  {x*}. 


2.  Some  notations  and  preliminary  results 

In  this  section  we  introduce  the  notations  used  in  this  paper  and  we  prove 
some  results  on  the  computation  of  determinants  which  will  be  necessary  in 
order  to  derive  our  main  results. 

We  denote  by  ]Rmxn  the  space  of  m  x  n  real  matrices  with  m  >  n  and 
we  assume  that  A  €  ]Rmx”  with  rank  A  =  n.  The  transpose  of  A  is  denoted 
AT .  The  matrix  obtained  from  A  by  replacing  column  k  with  x  £  JRm  is 
denoted  by: 

A(k  — ♦  x). 

The  minor  of  A  of  order  p  corresponding  to  rows  indexed  by  ij  and  columns 
indexed  by  kj  is  denoted  by: 


(  *i  h.-.ip  \ 

V  kt---kp  ) 

provided  1  <  t’x  <  *2  <  . . .  <  ip  <  m  and  1  <  k\  <  ki  <  .  .  .  <  kp  <  n.  If  A 
is  a  square  matrix  (m  =  n)then  the  determinant  of  A  is: 

1  2...n\ 

1  2  . . .  n  j 


detA  =  A 


An  important  formula  for  computing  the  determinant  of  the  product  of 
two  rectangular  matrices  is  the  so  called  Binet-Cauchy  formula  see  e.g.  [2]. 
Let  D  €  and  E  €  R’nxn  be  given  matrices  with  m  >  n,  and  let 

C  :=  DE  i.e.  C  €  Rr,xn.  Then, 


detC  =  ^  D 


1  2  'n  )  e(  h  12  ) 

M  •  2  •  ■  •  *  n  )  \  1  2  .  .  .n  J 


(2.1) 


A  direct  application  of  the  Binet-Cauchy  formula  leads  to  the  following 
result. 
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Lemma  2.1  Let  A  <E  IRmxn .  Then, 

det(AAT)  =  J3  A(*l 

Proof.  Let  D  =  aF  and  E  —  A  in  (2.1).  Then, 


detC  =  det(AAT )  =  5  ^  Ar 


*1  *2 •  •  •  *n 

1  2  .  .  .  n 


Since 

Ar 

the  result  follows.  □ 


1  2  .  .  .  n 

*1  *2  -  •  «n 


2. 

■•n.  )a 

( v  *2- 

•  •  *n 

*2  • 

v  1  2- 

.  n 

*1 

*2  •  •  -  ‘ft  \ 

1 

2  . . .  n  / 

Remark  2.1  An  alternative  proof  of  Lemma  2.1  is  given  by  Linnik,  [5,  p. 
25], 

The  next  result  is  a  simple  multiplication  rule  between  two  specific  matrices. 
Lemma  2.2  For  all  b  €  Rm  and  A  €  Rm*n 

ATA(k  — >  ATb)  =  Ar  •  A(k  — ♦  6)1 

Proof.  We  prove  the  result  for  fc  =  1.  Suppose  that  column  1  is  replaced 
in  AT A  by  Arb ,  then  the  first  column  of  Ar A  is 

m 

53  arpbr>  P  =  1j  •  •  • )  n 

r=  1 

Now  replacing  column  1  by  b  in  A  implies  that  the  only  column  changed  in 
the  product  Ar-A(l  — »  6)  is  the  first  column,  which  is  simply  arsbr,  s 
1, . . . ,  n.  Hence, 

ArA(l  -»  ATb)  =  At  ■  A(1  -*  b ) 

Repeating  the  same  argument  for  each  column  k  =  1, . .  .m,  the  result  fol¬ 
lows.  □ 

In  the  sequel  we  will  need  the  following  notations.  Let  Arm,n  denote  all 
subset  of  n  integers  out  of  {1,2,...,  m},  (m  >  n),  i.e. 

Nm,n  =  {(*1  >  •  •  •  i  •  1  —  1 1  ^  •••  *n  5: 

‘The  notation  is  used  here  to  indicate  that  in  the  RHS  we  replace  column  k  with  the 
vector  b  in  the  matrix  A  only 
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We  order  the  members  of  Nmin  by  increasing  lexicographic  order.  Let  i; 

be  the  j-th  such  index  set,  where  j  G  Pm,n  {1,2,..., 

otherwise  specified,  all  the  summations  in  the  rest  of  the  paper  are  taken 
over  j  G  Pmn •  Using  our  new  notations,  Lemma  2.1  can  be  rewritten  as: 

A  :=  detAT  A  =  A ). 

where  A /.  is  the  determinant  of  the  n  x  n  submatrix  Aj}  (with  rows  indices 
in  Ij).  The  corresponding  subvector  of  b  G  lRm  is  denoted  b/j  G  IR” 

Lemma  2.3  Let  A  €  Stmxn  and  b  €  Rm.  Then, 

detAT  A  ( k  — +  ATb)  =  ]P  det  A/;  detA[.  ( k  — +  6/y) 

Proof.  From  Lemma  2.2  we  have: 

detATA(k  — *  ATb)  =  detAT  ■  A(k  —*  b) 

By  Binet-Cauchy  formula: 

detAT  ■  A(k  — *  b)  =  ^  det  A*.  •  detAi^k  — ►  bI}) 

and  since  detAj^  =  det  A /y,  the  Lemma  is  proved.  □ 

3.  Results 

Consider  the  system  of  linear  equations 

Ax  =  b  A  G  JR™*",  6  €  HT71,  m  >  n  (3.1) 

We  assume  that  A  is  full  column  rank,  i.e.  rankA  =  n.  For  each  index  set 
Ij  G  Nm,n  ( j  €  Pm,n )  there  is  an  associated  square  subsystem  of  n  linear 
equations  out  of  (3.1): 

Aj,x  =  bIj 

If  A /.  =  detAi}  -£■  0  this  system  has  a  unique  solution  which  we  denote  by 
y/y,  i-e. 

Ajjyjj  —  bj} ,  j  G  Pm<n  =  {j  G  Pm,n  '■  &lj  7^  0}  (3-2) 

Note  that  P*  n  ^  0  since  rankA  =  n. 


J  }  Unless 
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The  least  Euclidean  norm  solution  of  (3.1)  ( least  squares  solution  for 
short)  is  the  unique  solution  of  the  so  called  normal  equations:  (see  e.g.  [l]) 

ArAx  =  ATb  (3.3) 

The  uniqueness  follows  from  A  detAT  A  ^  0  which  is  implied  by  rank.4  = 
n.  The  next  result  gives  an  explicit  representation  of  the  least  squares 
solution  as  a  convex  combination  of  the  yi/s. 

Theorem  3.1  The  least  squares  solution  of  the  system  (8.1)  is  given  by 

x=  E  Xr,Vr, 


where  Xr  = 

‘j 


Moreover ,  X /  >  0,  and  V  .  p+  Xr  =  1.  Hence, 

3  jtl*  m.n  3 


x  G  conv  {yi-  :  j  &  P+  n) 

Proof.  Since  A  =  detAT  A  0,  the  solution  of  (3.3)  is  unique  and  is  given 
by  the  Cramer  rule  (componentwise) 


detAT  A(k  —*  ATb) 
detAT  A 


k  =  1, .  . . ,  n. 


(3.4) 


By  Lemma  2.3,  (3.4)  can  be  written  as: 


where 


detAi  •  detAi  {k  —*  bi 
Xk  =  - d^A - - 


A ^{b)  :=  detAi .(k  —  6/y) 


Defining  A/y  :=  A]  /A,  (3.5)  can  be  written  as: 


Xk 


-  E  *'r-2r. 

jeri. 


,  k  =  1, .  . . ,  n 


Consider  now  the  solution  of  the  linear  equation  (3.2): 

Ar,yi,  =  bis  for  j  €  P+<n 


(3  5) 


(3.6) 
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Using  Cramer  rule  ,  the  solution  is  given  componentwise  by 


t  _  - h',)  _  A‘,w  ,  , 

- deiXT,  A,,’ 

Combining  (3.6)  and  (3.7)  it  follows  that 

Xk  =  !.•••>« 

j€Pm.n 

Furthermore,  by  Lemma  2.1,  A  =  YljePm  „  =  Sj6p+  ^l}>  hence 

A? 

A/y  =  >  0,  and  A/;  =  1.  □ 

]€Pm.n 


(3.7) 


Theorem  3.1  can  be  generalized  to  the  solution  of  a  weighted  least-squares 
problem.  Let  M  £  RmXm  be  the  diagonal  matrix 


M  =  diag(/tti,.  ..,Mm) 

with  diagonal  elements  /y,  >  0,  i  =  1, . . . ,  m.  The  solution  of  the  weighted 
least-squares  problem  with  weight  matrix  M  is  obtained  by  solving  the  nor¬ 
mal  equation: 

AtMA  =  AT  Mb  (3.8) 

Since  M  is  positive  definite  (3.8)  can  be  written  as: 

ATMl/2M1/2A  =  ATMl/2M1/2b  (3.9) 

where 

M1/2  =diag(^,,...,VMm) 

Define  A  —  Ml^2A,  b  =  Ml/2b,  then  since  (M1/2)7  =  M1/2,  (3.9)  is 
equivalent  to 

ATAx  =  ATb  (3.10) 

i.e.  x  is  the  least  squares  solution  of  the  linear  equation 

Ax  —  b 


Applying  Theorem  3.1  we  thus  obtain 

x=  L  yi, 

.n 
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where  yi;  solves 


~Ah  yh  =  h- 


and 


~  _  Ml  det2Ai, 

l,~  A  Zdet2AI] 


(3.11) 


We  show  below  that  the  determinants  A/;  and  A  can  be  expressed  explicitly 
in  terms  of  A/;,  A  and  fij,  and  that  the  coefficients  A /.  can  be  expressed 
in  terms  of  A/;  and  p.j.  Therefore  the  least  squares  solution  of  the  weighted 
problem  can  be  obtained  directly  from  the  least  norm  solution  of  the  orig¬ 
inal  linear  equation  Ax  —  b  by  a  simple  scaling  of  the  convex  combination 
coefficients  A/;. 


Theorem  3.2  The  weighted  least  squares  solution  of  (3.8)  is  given  by 


x=  E  */yWy 

j€P+.„ 


where  yii  solves  (3.2),  and  where 


^1)  2  1  1  1  ^  ^m,n 

£;€P+„  *ijxi y 


with 


ni> ;=  n  h) 

j€P  +  .n 

In  particular,  x  G  conv  {yi}  :  j  G 
Proof.  By  definition 

A  =  M1/2A 


1/2 


and  thus 

*li  =  M}j2  Ati  (3.12) 

where  is  the  n  x  n  diagonal  matrix  with  elements  /ifc  >  0,  A;  G  /3,  j  G 

Pmn.  Using  (3.12)  we  compute: 

A/;  =  detM1/2  ■  detA[; 

=  ( n  »)>/, 

j€Pm.n 
=  *li*Ii- 
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Substituting  the  above  in  (3.11)  and  using  A /  as  given  in  Theorem  3.1  we 
obtain 

_2  a  2 

~  nI 

\  _  iJ  iJ 


*ix'1 


(3.13) 


Now  using  (3.7)  we  have  for  k  =  1, . .  . ,  n 

_  Aji  (6)  _  A*  (6) 


y'i,= 


by  (3.13) 


(314) 


where 


A^(6)  =  detAj.  (k  —*  bi} 

=  detMjj2 Ai-(k -+  M^2b[.) 
=  detMj i2  ■  detAji  ( k  — *  6/;) 
=  »/,•*£(*) 


(3.15) 

(3.16) 


The  equality  (3.15)  following  from  the  fact  that  for  any  diagonal  matrix 
D  £  Rnxn  and  any  matrix  F  £  Rnxn  and  vector  c  £  R"  the  following 
identity  holds: 

DF{k  -  0c)  =  £>•  F(fc  —  c). 

Substituting  (3.16)  in  (3.14)  it  follows  that  y*  =  y*.,  for  all  A:  =  l,...,n 
and  the  proof  is  complete.  □ 

Remark  3.1  The  important  fact  to  observe  is  that  the  yj. ’ s  depend  only 
on  the  data  A,b  and  not  on  the  matrix  M,  hence  the  weighted  least  squares 
solution  is  inside  the  compact  polyhedral  convex  set  conv  (yj  :  j  6  P+n} 
which  is  independent  of  the  weighting  matrix  M. 
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